Storage medium storing image processing program and image processing apparatus

ABSTRACT

An image is placed, based on a predetermined designation input, on a predetermined placement position in relation to a display area, and displayed. The image is enlarged and reduced in accordance with an enlargement ratio of the image, and an area within which the placement position can be set is changed in relation to the display area in accordance with the enlargement ratio. Then, based on an obtained designation input, a position within the area, which position corresponds to the designation input, is calculated as a placement position, and the image having been enlarged and reduced is placed on the placement position. The image is then displayed on a display device.

CROSS REFERENCE TO RELATED APPLICATION

The disclosure of Japanese Patent Application No. 2007-257821, Oct. 1,2007, is incorporated herein by reference.

BACKGROUND OF THE INVENTION

1. Field of the Invention

The present invention relates to a storage medium storing an imageprocessing program and an image processing apparatus. The presentinvention particularly relates to a storage medium, which stores animage processing program for placing and displaying an image on adesignated placement position, and an image processing apparatus.

2. Description of the Background Art

Conventionally, an image processing apparatus for enlarging/reducing animage in accordance with a distance between an input device and imagingtargets (markers) is disclosed by Japanese Laid-Open Patent PublicationNo. 2007-236697 (hereinafter, referred to as Patent Document 1), forexample. In the image processing apparatus disclosed by Patent Document1, a controller has an image-pickup device for taking an image of twomarkers, and based on a distance between the two markers in the takenimage, the distance between the input device and imaging targets (i.e.,the markers) is calculated. Then, an image is: enlarged/reduced based onthe calculated distance; placed on a position which corresponds tocoordinates of a middle point between the two markers (e.g., a positionpointed by the controller); and then displayed on a display device.

However, as shown in FIGS. 20A and 20B, in the case of the imageprocessing apparatus disclosed by Patent Document 1, operabilitydeteriorates when an area within which an image can be moved is fixedfor images respectively having different enlargement ratios. In FIG.20A, an area within which coordinates Pims, which indicate a position ofan image IMs having a relatively low enlargement ratio, can be moved(movable area) is set. In the example of FIG. 20A, the movable area isset such that when the image IMs moves to a farthest edge of the movablearea, almost the entire image IMs is displayed on a display screen, anda part of the image IMs is outside the display screen.

In FIG. 20B, similarly to the case of the image IMs, a movable area ofcoordinates Pimb is set for an image IMb having a relatively highenlargement ratio. In this case, the movable area is set such that whenthe image IMb moves to a farthest edge of the movable area, ¼ of theimage IMs is displayed within the display area, and ¾ of the image IMsis outside the display area. Thus, in the case where the image IMbhaving a relatively high enlargement ratio is moved, as compared to thecase where the image IMs is moved, when the image IMb is moved to thefarthest edge of the movable area, a large portion of the image IMbremains within the display screen, and therefore it is likely that auser wishes to move the image IMb further to the outside of the movableare. However, such a wish of the user cannot be met since the movablearea is fixed, and this results in operability deterioration for theuser.

In the case of setting, assuming that the image IMb is to be moved, themovable area so as to extend to the outside of the display area, it isconceivable that if the image IMs having a relatively low enlargementratio is moved within the movable area having extended to the outside ofthe display area, the image IMs disappears from the display area, and isnot displayed on the display screen. In such a case, the user is unableto know where the image IMs is positioned, and this causes deteriorationin operability.

Further, in the case where the image, which is placed on the positioncorresponding to the coordinates of the middle point between the twomarkers, is displayed on the display device, there is a possibility thatthe position to be designated varies depending on the distance betweenthe controller and the markers. To be specific, in the case where thecontroller is operated at a remote position from the markers, even if adirection of the image-pickup device of the controller (i.e., directionin which the controller is pointed) is slightly changed, the designatedposition changes greatly, as compared to a case where the controller isoperated near the markers. In other words, even if the user performs asame operation by using the controller, a result of the operation variesdepending on the distance from the markers.

SUMMARY OF THE INVENTION

Therefore, an object of the present invention is to solve at least oneof the above-described problems, and to provide a storage medium storingan image processing program and an image processing apparatus which arecapable of properly controlling a movement of an image in accordancewith an enlargement ratio of the image or the distance between theimage-pickup device and the imaging targets.

The present invention has the following features to achieve the objectmentioned above. Note that reference numerals, step numbers (here,“step” is abbreviated as “S”), diagram numbers and the like indicatedbetween parentheses are merely provided to facilitate the understandingof the present invention in relation to the later-described best modeembodiment, rather than limiting the scope of the present invention inany way.

A first aspect of the present invention is a computer-readable storagemedium storing an image processing program to be executed by a computer(10) of an image processing apparatus (5) for: enlarging or reducing animage (IM); placing the image on a predetermined placement position(Pim) in relation to a display area in accordance with a predetermineddesignation input (Da; (Lx, Ly), (Rx, Ry)); and displaying the image.The image processing program causes the computer to function asdesignation input obtaining means (CPU 10 performing steps 50 and 64;hereinafter, only step numbers are indicated), enlarging and reducingmeans (S55, S60), area setting means (S57), placement positioncalculation means (S58), image placing means (S61), and display controlmeans (S62). The designation input obtaining means obtains thedesignation input. The enlarging and reducing means sets an enlargementratio (Sx, Sy) of the image, and enlarges and reduces the image inaccordance with the enlargement ratio. In accordance with theenlargement ratio (STx and STy calculated based on Sx and Sy) set by theenlarging and reducing means, the area setting means changes, inrelation to the display area, an area within which the placementposition is allowed to be set. The placement position calculation meanscalculates, based on the designation input obtained by the designationinput obtaining means, a position within the area as the placementposition, the position corresponding to the designation input. The imageplacing means places, on the placement position, the image enlarged andreduced by the enlarging and reducing means. The display control meanscauses a display device (2) to display the image placed by the imageplacing means. Note that, the area, within which the placement positionis allowed to be set and which is set on the display area, may be in thesame size as the display area, or smaller than the display area, orlarger than the display area.

In a second aspect of the present invention based on the first aspect,the area setting means sets the area so as to extend in accordance withan increase in the enlargement ratio set by the enlarging and reducingmeans.

In a third aspect of the present invention based on the second aspect,the designation input obtaining means obtains the designation input froman input device (7) having an image-pickup device (74) for taking animage of a predetermined imaging target (8). The image processingprogram further causes the computer to function as distance calculationmeans (S52, S65) for calculating a distance (D) between the imagingtarget and the image-pickup device. The enlarging and reducing meansincreases the enlargement ratio in accordance with a decrease in thedistance. The placement position calculation means calculates theplacement position based on the enlargement ratio and a position of theimaging target in the image taken by the image-pickup device.

In a fourth aspect of the present invention based on the third aspect,the placement position calculation means calculates the placementposition in the area set by the area setting means, by changing, basedon the enlargement ratio, a rate of change of the placement position inrelation to a change of the position of the imaging target in the takenimage.

In a fifth aspect of the present invention based on the third aspect, aplurality of imaging targets are provided. The distance calculationmeans calculates a distance between the image-pickup device and theimaging targets, based on a distance (mi) between the imaging targets inthe taken image.

In a sixth aspect of the present invention based on the fifth aspect,the computer is further caused to function as orientation calculationmeans (S54) and image orientation determination means (S59). Theorientation calculation means calculates an orientation of the inputdevice, based on a gradient (Db1) between the imaging targets in thetaken image. The image orientation determination means determines anorientation (Aim) of the image in accordance with the orientation of theinput device. The image placing means places, in the orientationdetermined by the image orientation determination means, the imageenlarged and reduced by the enlarging and reducing means.

In a seventh aspect of the present invention based on the first aspect,the area setting means uses an aspect ratio of the image, therebychanging an aspect ratio of the area which is changed in accordance withthe enlargement ratio.

In an eighth aspect of the present invention based on the seventhaspect, the computer is further caused to function as image orientationdetermination means for determining an orientation of the image inaccordance with the designation input. The area setting means uses theaspect ratio and the orientation of the image, thereby changing theaspect ratio of the area which is changed in accordance with theenlargement ratio.

A ninth aspect of the present invention is a computer-readable storagemedium storing an image processing program to be executed by a computerof an image processing apparatus for: placing the image on apredetermined placement position in relation to a display area inaccordance with a designation input obtained from an input device havingimage-pickup means for taking an image of an imaging target; anddisplaying the image. The image processing program causes the computerto function as designation input obtaining means, distance calculationmeans, placement position calculation means, image placing means anddisplay control means. The designation input obtaining means obtains thedesignation input from the input device. The distance calculation meanscalculates, based on a position of the imaging target in the takenimage, a distance between the imaging target and the image-pickup means.Based on the distance, the placement position calculation means changesa rate of change of the placement position in relation to a change ofthe position of the imaging target in the taken image, therebycalculating the placement position. The image placing means places theimage on the placement position. The display control means causes adisplay device to display the image placed by the image placing means.

In a tenth aspect of the present invention based on the ninth aspect,the placement position calculation means calculates the placementposition in such a manner as to increase the rate of change of theplacement position in accordance with a decrease in the distancecalculated by the distance calculation means.

In an eleventh aspect of the present invention based on the tenthaspect, the computer is further caused to function as area setting meansfor, in accordance with the distance calculated by the distancecalculation means, changing, in relation to the display area, an areawithin which the placement position is allowed to be set. The areasetting means sets the area so as to extend in accordance with adecrease in the distance calculated by the distance calculation means.

In a twelfth aspect of the present invention based on the ninth aspect,a plurality of imaging targets are provided. The distance calculationmeans calculates a distance between the image-pickup means and theimaging targets, based on a distance between the imaging targets in thetaken image.

A thirteenth aspect of the present invention is an image processingapparatus for: enlarging or reducing an image; placing the image on apredetermined placement position in a display area of a display devicein accordance with a predetermined designation input; and displaying theimage. The image processing apparatus comprises designation inputobtaining means, enlarging and reducing means, area setting means,placement position calculation means, image placing means and displaycontrol means. The designation input obtaining means obtains thedesignation input. The enlarging and reducing means sets an enlargementratio of the image, and enlarges and reduces the image in accordancewith the enlargement ratio. In accordance with the enlargement ratio setby the enlarging and reducing means, the area setting means changes, inrelation to the display area, an area within which the placementposition is allowed to be set. The placement position calculation meanscalculates, based on the designation input obtained by the designationinput obtaining means, a position within the area as the placementposition, the position corresponding to the designation input. The imageplacing means places, on the placement position, the image enlarged andreduced by the enlarging and reducing means. The display control meanscauses a display device to display the image placed by the image placingmeans.

In a fourteenth aspect of the present invention based on the thirteenthaspect, the area setting means sets the area so as to extend inaccordance with an increase in the enlargement ratio set by theenlarging and reducing means.

In a fifteenth aspect of the present invention based on the fourteenthaspect, the designation input obtaining means obtains the designationinput from an input device having an image-pickup device for taking animage of a predetermined imaging target. The image processing apparatusfurther comprises distance calculation means for calculating a distancebetween the imaging target and the image-pickup device. The enlargingand reducing means increases the enlargement ratio in accordance with adecrease in the distance. The placement position calculation meanscalculates the placement position based on the enlargement ratio and aposition of the imaging target in the image taken by the image-pickupdevice.

A sixteenth aspect of the present invention is an image processingapparatus for: placing an image on a predetermined placement position inrelation to a display area of a display device in accordance with adesignation input obtained from an input device having image-pickupmeans for taking an image of an imaging target; and displaying theimage. The image processing apparatus comprises designation inputobtaining means, distance calculation means, placement positioncalculation means, image placing means and display control means. Thedesignation input obtaining means obtains the designation input from theinput device. The distance calculation means calculates, based on aposition of the imaging target in the taken image, a distance betweenthe imaging target and the image-pickup means. The placement positioncalculation means changes, based on the distance, a rate of change ofthe placement position in relation to a change of the position of theimaging target in the taken image, thereby calculating the placementposition. The image placing means places the image on the placementposition. The display control means causes a display device to displaythe image placed by the image placing means.

In a seventeenth aspect of the present invention based on the sixteenthaspect, the placement position calculation means calculates theplacement position in such a manner as to increase the rate of change ofthe placement position in accordance with a decrease in the distancecalculated by the distance calculation means.

In an eighteenth aspect of the present invention based on the thirteenthaspect, the image processing apparatus further comprises area settingmeans for, in accordance with the distance calculated by the distancecalculation means, changing, in relation to the display area, an areawithin which the placement position is allowed to be set. The areasetting means sets the area so as to extend in accordance with adecrease in the distance calculated by the distance calculation means.

According to the above first aspect, the area within which the image canbe moved (i.e., movable area) is set in accordance with the enlargementratio of the image. This allows the movable area of the image to be setappropriately in accordance with the size of the image, and allows themovement of the image to be properly controlled in accordance with theenlargement ratio of the image.

According to the above second aspect, operability deterioration, such asthe movable area becoming insufficient for an image having a highenlargement ratio, is prevented. Further, for an image having a lowenlargement ratio, i.e., for an image displayed in a reduced size, themovable area is also in a reduced size. This prevents the image fromdisappearing from the display area.

According to the above third aspect, when the input device is movedtoward the imaging target, the displayed image is enlarged, and the areawithin which the image can be moved is also enlarged. On the other hand,when the input device is moved away from the imaging target, thedisplayed image is reduced, and the area within which the image can bemoved is also reduced. Thus, the image enlargement/reduction process canbe performed only by changing the distance between the input device andimaging target, that is, the enlargement ratio of the displayed imagecan be changed by a simple operation. Also, the displayed image moves inresponse to the input device having been moved from side to side and upand down with respect to the imaging target. Thus, the displayed imagecan be moved by a simple operation.

According to the above fourth aspect, when the image is to be displayedat a position pointed by the input device, a designated moving amount ofthe image is greater in the case where the input device is positionednear the imaging target than in the case where the input device is at aremote position from the imaging target. As a result, increase/decrease,caused by a difference in distance between the input device and imagingtarget, of the moving amount of the image in relation to an amount ofchange in the position pointed by the input device, is lessened. Inother words, for a user using the input device to perform an operationto move the pointed position, a significant change in operability doesnot occur even if the distance between the input device and the imagingtarget changes. This allows the user to perform the operation withoutfeeling discomfort.

According to the above fifth aspect, the distance between theimage-pickup device and the imaging targets can be easily calculated byusing a distance between at least two imaging targets in the image takenby the input device.

According to the above sixth aspect, by performing such an operation asto twist the input device to the right and left, the user can cause thedisplayed image to rotate in accordance with the operation.

According to the above seventh aspect, the area within which the imagecan be moved is set based on the aspect ratio of the image. As a result,the movable area is set appropriately in accordance with a shape of theimage, e.g., the image having a vertically long shape or a horizontallylong shape.

According to the above eighth aspect, the area within which the imagecan be moved is set in accordance with the orientation of the image. Asa result, the movable area can be set appropriately in accordance with,e.g., a change in the placement angle of the vertically-long-shaped orhorizontally-long-shaped image, i.e., in accordance with the shape andplacement angle of the image.

According to the above ninth aspect, the increase/decrease, caused by adifference in distance between the input device and imaging target, ofthe moving amount of the image in relation to the amount of change inthe position pointed by the input device, can be adjusted. For example,when the image is to be displayed on the position pointed by the inputdevice, even if an operation to move the pointed position is performedin a same manner, the moving amount of the pointed position changesdepending on the distance between the input device and imaging target.Then, the moving amount of the image, which moves in response to themovement of the pointed position, increases/decreases accordingly.However, by adjusting the moving amount of the image in accordance withthe distance between the input device and imaging target, operabilityrelated to the moving amount of the image, which moving amount is inresponse to an operation performed by the user to move the pointedposition by using the input device, can be adjusted. This allows theuser to perform the operation without feeling discomfort.

According to the above tenth aspect, when the image is to be displayedat the position pointed by the input device, a designated moving amountof the image is greater in the case where the input device is positionednear the imaging target than in the case where the input device is at aremote position from the imaging target. As a result, theincrease/decrease, caused by a difference in distance between the inputdevice and imaging target, of the moving amount of the image in relationto the amount of change in the position pointed by the input device, islessened. In other words, for the user using the input device to performan operation to move the pointed position, a significant change inoperability does not occur even if the distance between the input deviceand the imaging target changes. This allows the user to perform theoperation without feeling discomfort.

According to the above eleventh aspect, the area, within which the imagecan be moved, can be adjusted in size in accordance with the distancebetween the input device and imaging target, and the movable area can beenlarged in accordance with a decrease in the distance between the inputdevice and imaging target.

According to the above twelfth aspect, the distance between theimage-pickup device and the imaging targets can be easily calculated byusing a distance between at least two imaging targets in the image takenby the input device.

According to the image processing apparatus of the present invention,the same effect as that provided by the above-described storage mediumstoring the image processing program can be obtained.

These and other objects, features, aspects and advantages of the presentinvention will become more apparent from the following detaileddescription when taken in conjunction with the accompanying drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is an external view illustrating a game system 1 according to anembodiment of the present invention;

FIG. 2 is a functional block diagram of a game apparatus body 5 of FIG.1;

FIG. 3 is an isometric view of the controller 7 of FIG. 1, seen from atop rear side thereof;

FIG. 4 is an isometric view of the controller 7 of FIG. 3, seen from abottom front side thereof;

FIG. 5 is an isometric view illustrating a state where an upper casingof the controller 7 of FIG. 3 is removed;

FIG. 6 is an isometric view illustrating a state where a lower casing ofthe controller 7 of FIG. 4 is removed;

FIG. 7 is a block diagram showing a configuration of the controller 7 ofFIG. 3;

FIG. 8 shows an exemplary state of a player holding the controller 7with his/her right hand as seen from a front surface side of thecontroller 7;

FIG. 9 shows an exemplary state of the player holding the controller 7with his/her right hand as seen from a left side of the controller 7;

FIG. 10 illustrates a viewing angle between an imaging informationcalculation section 74 and markers 8L and 8R;

FIG. 11 is a top view showing an example in which a player P operatesthe controller 7 in a front-rear direction with respect to the markers8L and 8R;

FIG. 12 is an exemplary screen display which is provided on the monitor2 in response to the operation shown in FIG. 11;

FIG. 13 is an exemplary screen display which is provided on the monitor2 in response to the operation shown in FIG. 11;

FIG. 14 shows operation angles in the case where the player P pointsboth ends of a display screen of the monitor 2 by using the controller7;

FIG. 15 shows an example of main data stored in a main memory of thegame apparatus body 5;

FIG. 16 is a flowchart showing a flow of image processing performed bythe game apparatus body 5;

FIG. 17 shows a subroutine showing in detail a distance calculationprocess at steps 52 and 65 of FIG. 16;

FIG. 18 is a diagram for describing an operation to calculate a distanceD;

FIG. 19 is a conceptual diagram showing an image position and movablearea set in a virtual game space;

FIG. 20A shows a conventional movable area for an image having arelatively low enlargement ratio; and

FIG. 20B shows a conventional movable area for an image having arelatively high enlargement ratio.

DESCRIPTION OF THE PREFERRED EMBODIMENTS

With reference to FIG. 1, an apparatus for executing an image processingprogram according to an embodiment of the present invention will bedescribed. Hereinafter, in order to give a specific description, a gamesystem including a stationary game apparatus body 5 which is an exampleof the above apparatus will be described. FIG. 1 is an external viewillustrating a game system 1 including a stationary game apparatus 3.FIG. 2 is a block diagram of the game apparatus body 5. The game system1 will be described below.

As shown in FIG. 1, the game system 1 comprises: a home-use TV receiver2 (hereinafter, referred to as a monitor 2) which is an example ofdisplay means; and the stationary game apparatus 3 connected to themonitor 2 via a connection cord. The monitor 2 has speakers 2 a foroutputting, in the form of sound, an audio signal outputted from thegame apparatus body 5. The game apparatus 3 includes: an optical disc 4in which a game program, which is an exemplary image processing programof the present invention, is stored; the game apparatus body 5 having acomputer for executing the game program of the optical disc 4, therebycausing the monitor 2 to output a game display; and a controller 7 forproviding the game apparatus body 5 with necessary operation informationfor a game in which a character or the like displayed in the gamedisplay is controlled.

The game apparatus body 5 has a wireless controller module 19 therein(see FIG. 2). The wireless controller module 19 receives data wirelesslytransmitted from the controller 7, and transmits data from the gameapparatus body 5 to the controller 7. In this manner, the controller 7and game apparatus body 5 are connected by radio communication. Further,the optical disc 4 as an example of an exchangeable information storagemedium is detachably mounted on the game apparatus body 5.

On the game apparatus body 5, a flash memory 17 (see FIG. 2) is mounted,the flash memory 17 functioning as a backup memory for fixedly storing,e.g., later-described various data and saved data for game softwareprocessing. The game apparatus body 5 executes the game program or thelike stored on the optical disc 4, and displays a result thereof as agame image on the monitor 2. The game program or the like to be executedmay be stored not only on the optical disc 4, but also prestored in theflash memory 17. The game apparatus body 5 can reproduce a state of thegame played in the past, by using the saved data stored in the flashmemory 17, and display a game image of the reproduced state on themonitor 2. A player playing with the game apparatus body 5 can enjoy thegame by operating the controller 7 while watching the game imagedisplayed on the monitor 2.

By using the technology of, for example, Bluetooth (registeredtrademark), the controller 7 wirelessly transmits transmission data suchas operation information to the game apparatus body 5 having thewireless controller module 19 therein. The controller 7 is operationmeans for mainly controlling an object or the like displayed on adisplay screen of the monitor 2. The controller 7 has a housing, whichis small enough to be held by one hand, and a plurality of operationbuttons (including a cross key, a stick or the like) exposed at asurface of the housing. As described later in detail, the controller 7includes an imaging information calculation section 74 for taking animage of a view seen from the controller 7. As exemplary imaging targetsof the imaging information calculation section 74, two LED modules 8Land 8R (hereinafter, referred to as “markers 8L and 8R”) are provided inthe vicinity of the display screen of the monitor 2. The markers 8L and8R each output, e.g., an infrared light forward from the monitor 2. Thecontroller 7 is capable of receiving, at a communication section 75,transmission data wirelessly transmitted from the wireless controllermodule 19 of the game apparatus body 5, and generating a sound orvibration based on the transmission data.

Next, an internal configuration of the game apparatus body 5 will bedescribed with reference to FIG. 2. FIG. 2 is a block diagram showingthe configuration of the game apparatus body 5. The game apparatus body5 has a CPU (Central Processing Unit) 10, system LSI (Large ScaleIntegration) 11, external main memory 12, ROM/RTC (Read Only Memory/RealTime Clock) 13, disc drive 14, AV-IC (Audio Video-Integrated Circuit)15, and the like.

The CPU 10 performs game processing by executing the game program storedin the optical disc 4, and acts as a game processor. The CPU 10 isconnected to the system LSI 11. In addition to the CPU 10, the externalmain memory 12, ROM/RTC 13, disc drive 14 and the AV-IC 15 are connectedto the system LSI 11. The system LSI 11 performs processing such as:controlling data transfer among components connected to the system LSI11; generating an image to be displayed; and obtaining data fromexternal devices. An internal configuration of the system LSI 11 will bedescribed later. The external main memory 12, which is a volatilememory, stores such a program as the game program loaded from theoptical disc 4, or the game program loaded from the flash memory 17, andvarious data. The external main memory 12 is used as a work region orbuffer region of the CPU 10. The ROM/RTC 13 has a ROM, in which a bootprogram for the game apparatus body 5 is incorporated (so-called bootROM), and a clock circuit (RTC) which counts the time. The disc drive 14reads program data, texture data and the like from the optical disc 4,and writes the read data into a later-described internal main memory 35or into the external main memory 12.

On the system LSI 11, an input/output processor 31, a GPU (GraphicProcessor Unit) 32, a DSP (Digital Signal Processor) 33, a VRAM (VideoRAM) 34 and the internal main memory 35 are provided. Although notshown, these components 31 to 35 are connected to each other via aninternal bus.

The GPU 32 partly forms rendering means, and generates an image inaccordance with a graphics command from the CPU 10. The VRAM 34 storesnecessary data for the GPU 32 to execute the graphics command (data suchas polygon data and texture data). At the time of generating the image,the GPU 32 uses the data stored in the VRAM 34, thereby generating imagedata.

The DSP 33 acts as an audio processor, and generates audio data by usingsound data and sound waveform (tone) data stored in the internal mainmemory 35 and external main memory 12. In order for the speakers 2 a tooutput a sound, the DSP 33 reads the sound data, and causes the speakers2 a of the monitor 2 to output the sound data via the AV-IC 15 and an AVconnector 16. In order for a speaker 706 (see FIG. 7) of the controller7 to output a sound, the DSP 33 reads the sound data, and transmits theaudio data to the controller 7 via the wireless controller module 19 andan antenna 23.

The image data and audio data generated in the above manner are read bythe AV-IC 15. The AV-IC 15 outputs the read image data to the monitor 2via the AV connector 16, and outputs the read audio data to the speakers2 a embedded in the monitor 2. As a result, the image is displayed onthe monitor 2 and the sound is outputted from the speakers 2 a.

The input/output processor (I/O Processor) 31 performs datatransmission/reception with components connected thereto, and downloadsdata from external devices, for example. The input/output processor 31is connected to the flash memory 17, a wireless communication module 18,the wireless controller module 19, an expansion connector 20 and anexternal memory card connector 21. An antenna 22 is connected to thewireless communication module 18, and the antenna 23 is connected to thewireless controller module 19.

The input/output processor 31 is connected to a network via the wirelesscommunication module 18 and antenna 22, thereby communicating with othergame apparatuses and various servers connected to the network. Theinput/output processor 31 regularly accesses the flash memory 17 todetect presence or absence of data which is required to be transmittedto the network. If such data is present, the data is transmitted to thenetwork via the wireless communication module 18 and antenna 22. Also,the input/output processor 31 receives, via the network, antenna 22 andwireless communication module 18, data transmitted from ether gameapparatuses or data downloaded from a download server, and stores thereceived data in the flash memory 17. By executing the game program, theCPU 10 reads the data stored in the flash memory 17 to use the data forexecution of the game program and image processing program. In additionto the data transmitted and received between the game apparatus body 5and other game apparatuses or various servers, the flash memory 17 maystore, as described above, saved data of the game which is played usingthe game apparatus body 5 (such as result data or progress data of thegame).

Further, the input/output processor 31 receives, via the antenna 23 andwireless controller module 19, operation data or the like transmittedfrom the controller 7, and stores (temporarily) the operation data orthe like in a buffer region of the internal main memory 35 or externalmain memory 12. Note that, similarly to the external main memory 12, theinternal main memory 35 may store such a program as the game programloaded from the optical disc 4, or the game program loaded from theflash memory 17, and various data. The internal main memory 35 may beused as a work region or buffer region of the CPU 10.

In addition, the expansion connector 20 and the external memory cardconnector 21 are connected to the input/output processor 31. Theexpansion connector 20 is a connector for such interface as USB or SCSI.The expansion connector 20, instead of the wireless communication module18, is able to perform communications with a network by being connectedto such a medium as external storage medium, a peripheral device, e.g.,another controller, or a connector for wired communication. The externalmemory card connector 21 is a connector to be connected to an externalstorage medium such as a memory card. For example, the input/outputprocessor 31 is able to access the external storage medium via theexpansion connector 20 or external memory card connector 21 to store orread data from the external storage medium.

On the game apparatus body 5 (e.g., on a front main surface), a powerbutton 24 of the game apparatus body 5, a reset button 25 for gameprocessing, an insertion slot for mounting the optical disc 4 in adetachable manner, an eject button 26 for ejecting the optical disc 4from the insertion slot of the game apparatus body 5, and the like areprovided. The power button 24 and reset button 25 are connected to thesystem LSI 11. When the power button 24 is turned on, each component ofthe game apparatus body 5 is supplied with power via an AC adaptor whichis not shown. When the reset button 25 is pressed, the system LSI 11reexecutes the boot program of the game apparatus body 5. The ejectbutton 26 is connected to the disc drive 14. When the eject button 26 ispressed, the optical disc 4 is ejected from the disc drive 14.

With reference to FIGS. 3 and 4, the controller 7 will be described.FIG. 3 is an isometric view of the controller 7 seen from a top rearside thereof. FIG. 4 is an isometric view of the controller 7 seen froma bottom front side thereof.

As shown in FIGS. 3 and 4, the controller 7 includes a housing 71 formedby plastic molding or the like. The housing 71 has a plurality ofoperation sections 72 provided thereon. The housing 71 has anapproximately parallelepiped shape extending in a longitudinal directionfrom front to rear. The overall size of the housing 71 is small enoughto be held by one hand of an adult or even a child.

At the center of a front part of a top surface of the housing 71, across key 72 a is provided. The cross key 72 a is a cross-shapedfour-direction push switch. The cross key 72 a includes operationportions corresponding to four directions (front, rear, right and left),which are respectively located on cross-shaped projecting portionsarranged at intervals of 90 degrees. A player selects one of the front,rear, right and left directions by pressing one of the operationportions of the cross key 72 a. Through an operation of the cross key 72a, the player can, for example, indicate a direction in which a playercharacter or the like appearing in a virtual game world is to move, orgive an instruction to select one of a plurality of options.

The cross key 72 a is an operation section for outputting an operationsignal in accordance with the above-described direction input operationperformed by the player. Such an operation section may be provided inanother form. For example, an operation section, which has four pushswitches arranged in crisscross directions and which is capable ofoutputting an operation signal in accordance with a push switch pressedby the player, may be provided. Alternatively, an operation section,which has a composite switch having, in addition to the above four pushswitches, a center switch at an intersection point of the abovecrisscross directions, may be provided. Still alternatively, the crosskey 72 a may be replaced with an operation section which includes aninclinable stick (so-called joy stick) projecting from a top surface ofthe housing 71 and which outputs an operation signal in accordance withan inclining direction of the stick. Still alternatively, the cross key72 a may be replaced with an operation section which includes ahorizontally-slidable disc-shaped member and which outputs an operationsignal in accordance with a sliding direction of the disc-shaped member.Still alternatively, the cross key 72 a may be replaced with a touchpad.

Behind the cross key 72 a on the top surface of the housing 71, aplurality of operation buttons 72 b to 72 g are provided. The operationbuttons 72 b to 72 g are each an operation section for, when the playerpresses a head thereof, outputting a corresponding operation signal. Forexample, functions as a 1st button, 2nd button and A button are assignedto the operation buttons 72 b to 72 d. Also, functions as a minusbutton, home button and plus button are assigned to the operationbuttons 72 e to 72 g, for example. Various operation functions areassigned to the operation buttons 72 a to 72 g in accordance with thegame program executed by the game apparatus body 5. In an exemplaryarrangement shown in FIG. 3, the operation buttons 72 b to 72 d arearranged in a line at the center in a front-rear direction on the topsurface of the housing 71. The operation buttons 72 e to 72 g arearranged on the top surface of the housing 71 in a line in a left-rightdirection between the operation buttons 72 b and 72 d. The operationbutton 72 f has a top surface thereof buried in the top surface of thehousing 71, so as not to be inadvertently pressed by the player.

In front of the cross key 72 a on the top surface of the housing 71, anoperation button 72 h is provided. The operation button 72 h is a powerswitch for turning on and off the power to the game apparatus body 5 byremote control. The operation button 72 h also has a top surface thereofburied in the top surface of the housing 71, so as not to beinadvertently pressed by the player.

Behind the operation button 72 c on the top surface of the housing 71, aplurality of LEDs 702 are provided. Here, a controller type (number) isassigned to the controller 7 such that the controller 7 isdistinguishable from the other controllers 7. The LEDs 702 are used for,e.g., informing the player of the controller type which is currently setfor the controller 7. Specifically, a signal is transmitted from thewireless controller module 19 to the controller 7 such that one of theplurality of LEDs 702, which corresponds to the controller type of thecontroller 7, is lit up.

On the top surface of the housing 71, sound holes for outputting soundsfrom a later-described speaker (speaker 706 shown in FIG. 5) to theexternal space are formed between the operation button 72 b and theoperation buttons 72 e to 72 g.

On a bottom surface of the housing 71, a recessed portion is formed. Therecessed portion on the bottom surface of the housing 71 is formed in aposition in which an index finger or middle finger of the player islocated when the player holds the controller 7 so as to point a frontsurface thereof to the markers 8L and 8R. On a slope surface of therecessed portion, an operation button 72 i is provided. The operationbutton 72 i is an operation section acting as, for example, a B button.

On a front surface of the housing 71, an image pickup element 743forming a part of the imaging information calculation section 74 isprovided. The imaging information calculation section 74 is a systemfor: analyzing image data of an image taken by the controller 7;identifying an area having a high brightness in the image; and detectinga position of a center of gravity, a size and the like of the area. Theimaging information calculation section 74 has, for example, a maximumsampling period of approximately 200 frames/sec, and therefore can traceand analyze even a relatively fast motion of the controller 7. Aconfiguration of the imaging information calculation section 74 will bedescribed later in detail. On a rear surface of the housing 71, aconnector 73 is provided. The connector 73 is, for example, an edgeconnector, and is used for engaging and connecting the controller 7 witha connection cable.

Next, an internal structure of the controller 7 will be described withreference to FIGS. 5 and 6. FIG. 5 is an isometric view, seen from arear surface side of the controller 7, illustrating a state where anupper casing (a part of the housing 71) of the controller 7 is removed.FIG. 6 is an isometric view, seen from a front surface side of thecontroller 7, illustrating a state where a lower casing (a part of thehousing 71) of the controller 7 is removed. Here, FIG. 6 is an isometricview showing a reverse side of a substrate 700 shown in FIG. 5.

As shown in FIG. 5, the substrate 700 is fixed inside the housing 71. Ona top main surface of the substrate 700, the operation buttons 72 a to72 h, an acceleration sensor 701, the LEDs 702, an antenna 754 and thelike are provided. These elements are connected to, e.g., amicrocomputer 751 (see FIGS. 6 and 7) by wirings (not shown) formed onthe substrate 700 and the like. The wireless module 753 (see FIG. 7) andantenna 754 allow the controller 7 to act as a wireless controller.Inside the housing 71, a quartz oscillator, which is not shown, isprovided, and the quarts oscillator generates a reference clock of thelater-described microcomputer 751. Further, the speaker 706 and anamplifier 708 are provided on the top main surface of the substrate 700.

The acceleration sensor 701 is provided, on the substrate 700, to theleft side of the operation button 72 d (i.e., provided not on a centralportion but on a peripheral portion of the substrate 700). For thisreason, in response to the controller 7 having rotated around an axis ofa longitudinal direction of the controller 7, the acceleration sensor701 is able to detect, in addition to a change in direction ofgravitational acceleration, an acceleration containing centrifugalcomponents, and the game apparatus body 5 or the like is able todetermine, based on detected acceleration data, a motion of thecontroller 7 by a predetermined calculation with a favorablesensitivity. For example, the controller 7 has a three-axis accelerationsensor 701. The three-axis acceleration sensor 701 is able to detectlinear acceleration of the controller 7 for three axial directions ofthe controller 7, i.e., an up-down direction, a left-right direction,and a front-rear direction. Data indicating the acceleration detected bythe acceleration sensor 701 is outputted to the communication section75.

As shown in FIG. 6, at a front edge of a bottom main surface of thesubstrate 700, the imaging information calculation section 74 isprovided. The imaging information calculation section 74 comprises aninfrared filter 741, a lens 742, the image pickup element 743 and animage processing circuit 744 which are located in said order from thefront surface of the controller 7. These elements are attached to thebottom main surface of the substrate 700. At a rear edge of the bottommain surface of the substrate 700, the connector 73 is attached.Further, a sound IC 707 and the microcomputer 751 are provided on thebottom main surface of the substrate 700. The sound IC 707 is connectedto the microcomputer 751 and amplifier 708 by wirings formed on thesubstrate 700 and the like, and outputs audio signals via the amplifier708 to the speaker 706 in response to sound data transmitted from thegame apparatus body 5.

On the bottom main surface of the substrate 700, a vibrator 704 isattached. The vibrator 704 may be, for example, a vibration motor or asolenoid. The vibrator 704 is connected to the microcomputer 751 bywirings formed on the substrate 700 and the like, and is activated ordeactivated in response to vibration data transmitted from the gameapparatus body 5. The controller 7 is vibrated by an actuation of thevibrator 704, and the vibration is conveyed to the player's hand holdingthe controller 7. Thus, a so-called vibration-feedback game is realized.Since the vibrator 704 is provided at a relatively forward position inthe housing 71, the housing 71 held by the player significantlyvibrates, and allows the player to clearly feel the vibration.

Next, an internal configuration of the controller 7 will be describedwith reference to FIG. 7. FIG. 7 is a block diagram showing theconfiguration of the controller 7.

As shown in FIG. 7, in addition to the above-described operationsections 72, imaging information calculation section 74, accelerationsensor 701, vibrator 704, speaker 706, sound IC 707 and the amplifier708, the controller 7 includes the communication section 75.

The imaging information calculation section 74 includes the infraredfilter 741, lens 742, image pickup element 743 and the image processingcircuit 744. The infrared filter 741 allows, among lights incidentthereon through the front surface of the controller 7, only an infraredlight to pass therethrough. The lens 742 converges the infrared lightwhich has passed through the infrared filter 741, and outputs theinfrared light to the image pickup element 743. The image pickup element743 is a solid-state image pickup element such as a CMOS sensor or aCOD. The image pickup element 743 takes an image of the infrared lightcollected by the lens 742. In other words, the image pickup element 743takes an image of only the infrared light which has passed through theinfrared filter 741. Then, the image pickup element 743 generates imagedata of the image. The image data generated by the image pickup element743 is processed by the image processing circuit 744. Specifically, theimage processing circuit 744 processes the image data obtained from theimage pickup element 743, detects an area of the image, which area has ahigh brightness, and outputs, to the communication section 75, processresult data indicating, e.g., position coordinates, square measure andthe like detected from the area. The imaging information calculationsection 74 is fixed to the housing 71 of the controller 7. An imagingdirection of the imaging information calculation section 74 can bechanged by changing a facing direction of the housing 71.

The communication section 75 includes the microcomputer 751, a memory752, the wireless module 753 and the antenna 754. The microcomputer 751controls the wireless module 753 for wirelessly transmittingtransmission data while using the memory 752 as a storage area duringprocessing. The microcomputer 751 also controls operations of the soundIC 707 and vibrator 704 in accordance with data which the wirelessmodule 753 has received from the game apparatus body 5 via the antenna754. The sound IC 707 processes sound data or the like transmitted fromthe game apparatus body 5 via the communication section 75. Further, themicrocomputer 751 activates the vibrator 704 in accordance withvibration data or the like (e.g., a signal for causing the vibrator 704to be ON or OFF) which is transmitted from the game apparatus body 5 viathe communication section 75.

Data from the controller 7 such as operation signals (key data) from theoperation sections 72, acceleration signals (acceleration data) from theacceleration sensor 701 with respect to the three axial directions, andthe process result data from the imaging information calculation section74 are outputted to the microcomputer 751. The microcomputer 751temporarily stores inputted data (the key data, acceleration data andprocess result data) in the memory 752 as transmission data to betransmitted to the wireless controller module 19. Here, radiotransmission from the communication section 75 to the wirelesscontroller module 19 is performed at predetermined time intervals. Sincethe game processing is generally performed at a cycle of 1/60 sec, theradio transmission needs to be performed at a cycle of a shorter timeperiod. Specifically, the game processing is performed at a cycle of16.7 ms ( 1/60 sec), and a transmission interval of the communicationsection 75 structured using the Bluetooth (registered trademark)technology is 5 ms. At a timing of performing a transmission to thewireless controller module 19, the microcomputer 751 outputs, to thewireless module 753, the transmission data stored in the memory 752 as aseries of pieces of operation information. The wireless module 753 uses,for example, the Bluetooth (registered trademark) technology to radiate,with a carrier wave having a predetermined frequency, a radio signalfrom the antenna 754, the radio signal indicating the series of piecesof operation information. Thus, the key data from the operation sections72, the acceleration data from the acceleration sensor 701, and theprocess result data from the imaging information calculation section 74are transmitted from the controller 7. The wireless controller module 19of the game apparatus body 5 receives the radio signal, and the gameapparatus body 5 demodulates or decodes the radio signal to obtain theseries of pieces of operation information (the key data, accelerationdata and process result data). In accordance with the series of piecesof obtained operation information and the game program, the CPU 10 ofthe game apparatus body 5 performs game processing. In the case wherethe communication section 75 is structured using the Bluetooth(registered trademark) technology, the communication section 75 can havea function of receiving transmission data wirelessly transmitted fromother devices.

With reference to FIGS. 8 and 9, a state of a player holding thecontroller 7 with one hand will be described. FIG. 8 shows an exemplarystate of the player holding the controller 7 with his/her right hand asseen from a front surface side of the controller 7. FIG. 9 shows anexemplary state of the player holding the controller 7 with his/herright hand as seen from a left side of the controller 7.

As shown in FIGS. 8 and 9, the overall size of the controller 7 is smallenough to be held by one hand of an adult or even a child. When theplayer puts his/her thumb on the top surface of the controller 7 (forexample, near the cross key 72 a), and puts his/her index finger in therecessed portion on the bottom surface of the controller 7 (for example,near the operation button 72 i), a light entrance of the imaginginformation calculation section 74 on the front surface of thecontroller 7 is exposed forward from the player. It should be understoodthat also when the player holds the controller 7 with his/her left hand,the holding state is the same as that described for the right hand.

Thus, the controller 7 allows the player to easily operate the operationsections 72 such as the cross key 72 a and the operation button 72 iwhile holding the controller 7 with one hand. Further, when the playerholds the controller 7 with one hand, the light entrance of the imaginginformation calculation section 74 on the front surface of thecontroller 7 is exposed, whereby the light entrance can easily receivethe infrared lights from the aforementioned two markers 8L and 8R. As aresult, the player can hold the controller 7 with one hand withoutpreventing the imaging information calculation section 74 of thecontroller 7 from functioning. That is, when the player moves his/herhand holding the controller 7 with respect to the display screen, thecontroller 7 can perform an operation input by which a motion of theplayer's hand directly affects a display on the display screen.

As described above, in order to play a game with the game system 1 byusing the controller 7, the player holds the controller 7 with one hand(e.g., right hand). Here, the player holds the controller 7 such thatthe front surface of the controller (the entrance of a light whose imageis taken by the imaging information calculation section 74) faces themonitor 2. The two markers 8L and 8R are provided in the vicinity of thedisplay screen of the monitor 2 (see FIG. 1). These markers 8L and 8Reach output an infrared light forward from the monitor 2, and an imageof these infrared lights is taken by the imaging information calculationsection 74.

As shown in FIG. 10, the markers 8L and 8R each have a viewing angle θ1.The image pickup element 743 has a viewing angle θ2. For example, eachof the viewing angles θ1 of the markers 8L and 8R is 34 degrees(half-value angle), and the viewing angle θ2 of the image pickup element743 is 41 degrees. When both the markers 8L and 8R are present in theviewing angle θ2 of the image pickup element 743, and the image pickupelement 743 is present in the viewing angle θ1 of the marker BL and theviewing angle θ1 of the marker 8R, the game apparatus body 5 calculatesa position of the controller 7 (including the distance between thecontroller 7 and the markers 8L and 8R) by using positional data aboutpoints on the two markers 8L and 8R, the points each having a highbrightness.

When the player holds the controller 7 so as to point the front surfacethereof to the monitor 2, the infrared lights outputted from the twomarkers 8L and 8R are incident on the imaging information calculationsection 74. The image pickup element 743 takes an image of the infraredlights which are incident on the image pickup element 743 through theinfrared filter 741 and the lens 742, and the image processing circuit744 processes the taken image. The imaging information calculationsection 74 detects, from the taken image, infrared components outputtedby the markers 8L and 8R so as to obtain positional information aboutthe markers 8L and 8R (i.e., positions of target images) in the takenimage and size information about the markers 8L and 8R such as a squaremeasure, diameter and width thereof. Specifically, the image processingcircuit 744 analyzes image data of the image taken by the image pickupelement 743, and eliminates, from the taken image, an image which doesnot contain the infrared lights outputted by the markers 8L and 8R, andthen identifies points each having a high brightness as positions of themarkers 8L and 8R. The imaging information calculation section 74obtains positional information which is information about a brightnessposition such as the center of gravity of each of the identified pointshaving a high brightness, and outputs the positional information as theprocess result data. Here, the positional information outputted as theprocess result data may be coordinate values indicating the brightnessposition, which are obtained by setting a predetermined reference point(for example, the center or the upper left corner of the taken image) inthe taken image as a coordinate origin. Alternatively, the brightnessposition which is previously identified at a predetermined timing may beset as a reference point, and a vector indicating a positionaldifference between the reference point and the brightness positioncurrently identified may be outputted as the process result data. Thatis, in the case where a predetermined reference point is set in theimage taken by the image pickup element 743, the positional informationabout each of the target images in the taken image is a parameterindicating a positional difference from the predetermined referencepoint. When such positional information is transmitted to the gameapparatus body 5, the game apparatus body 5 can obtain, based on apositional difference between the reference point and the positionalinformation about each of the target images, an amount by which a signalchanges in accordance with a motion, orientation, position and the likeof the imaging information calculation section 74 (i.e., the controller7) with respect to the markers 8L and 8R. Specifically, the position ofeach point having a high brightness in the taken image, which istransmitted from the communication section 75, is changed in accordancewith the motion of the controller 7, and a direction or coordinatescorresponding to such a change of the position of each point having ahigh brightness is transmitted from the communication section 75. Uponreceiving the direction or coordinates from the communication section75, the game apparatus boy 5 recognizes and uses the direction orcoordinates as an input from the communication section 75, which inputcorresponds to a moving direction of the controller 7 in athree-dimensional space. In exemplary game processing described later,the imaging information calculation section 74 obtains at leastcoordinates of the center of gravity of a point having a high brightnessfor each of the target images of the markers 8L and 8R in the takenimage, and outputs the coordinates as the process result data.

Thus, the imaging information calculation section 74 of the controller 7takes an image of the stationary markers (infrared lights from the twomarkers 8L and 8R in the present embodiment), and the game apparatusbody 5 processes data outputted by the controller 7 during the gameprocessing. This enables an operation input to be performed inaccordance with the motion, orientation, position and the like of thecontroller 7. Therefore, the operation input, which is different from anoperation input made by pressing an operation button or using anoperation key, is intuitively performed. As described above, since themarkers are provided in the vicinity of the display screen of themonitor 2, the motion, orientation, position and the like of thecontroller 7 with respect to the display screen of the monitor 2 can beeasily calculated based on a position of the controller 7 with respectto the markers. That is, the process result data based on the motion,orientation, position and the like of the controller 7 can be used as anoperation input which directly affects a display on the display screenof the monitor 2. Note that, in the game system 1, the distance betweenthe controller 7 and the markers 8L and 8R, which is obtained by usingthe taken image of the markers 8L and 8R, can also be used as anoperation input which directly affects a display on the display screenof the monitor 2. This will be described later in detail.

Next, an exemplary image, which is displayed on the monitor 2 inaccordance with an operation performed by a player P, will be describedwith reference to FIGS. 11 to 13. FIG. 11 is a top view showing anexample in which the player P operates the controller 7 in a front-reardirection with respect to the markers 8L and 8R. FIGS. 12 and 13 showexemplary displays which are provided on the monitor 2 in accordancewith the operations, shown in FIG. 11, performed by the player P.

In FIG. 11, the player P holds the controller 7 so as to point the frontsurface of the controller 7 to the markers 8L and 8R (i.e., to themonitor 2). Here, a distance between the front surface of the controller7 (to be exact, the image pickup element 743) and a middle point betweenthe markers 8L and 8R is referred to as a “distance D”. In a state Ashown in FIG. 11, the player P holds the controller 7 such that thedistance D is a distance Dbs (i.e., later-described reference distanceDbs). The player P can change the distance D, which is the distancebetween the front surface of the controller 7 and the middle pointbetween the markers 8L and 8R, by moving the controller 7 back and forthwith respect to the monitor 2. For example, the player P may move thecontroller 7 forward to the monitor 2, such that the distance D changesfrom Dbs to Dn (state B). Also, the player P may move the controller 7backward from the monitor 2, such that the distance D changes from Dbsto Df (state C).

FIG. 12 is an exemplary screen display which is provided on the monitor2 in the case of the above state B. As shown in FIG. 12, the monitor 2displays, in the state B, a part of an image IMb which is superimposedon an background image. Here, the image IMb is a character image whichmoves and/or rotates within an display area of the monitor 2 in responseto the controller 7 pointing a particular position and/or in response toan orientation of the controller 7. The image IMb displayed on themonitor 2 in the state B is larger in size than the image IMb displayedin the state A. In other words, by moving the controller 7 toward themonitor 2, the player P can enlarge the character image (image IMb)displayed on the monitor 2. Moreover, by moving the controller 7 fromside to side and up and down with respect to the monitor 2 and/ortwisting the controller 7 to the right and left with respect to themonitor 2, the player P can move and/or rotate the image IMb displayedon the monitor 2. In FIG. 12, a position of the image IMb, which is setin accordance with a position which the player P points with thecontroller 7, is shown as an image position Pimb.

FIG. 13 is an exemplary screen display which is provided on the monitor2 in the case of the above state C. As shown in FIG. 13, the monitor 2displays, in the state C, an image IMs which is superimposed on anbackground image. Similarly to the image IMb, the image IMs is acharacter image which moves and/or rotates within the display area ofthe monitor 2 in response to the controller pointing a particularposition and/or in response to an orientation of the controller 7. Theimage IMs displayed on the monitor 2 in the state C is smaller in sizethan the image IMs displayed in the state A, or than the image IMb shownin FIG. 12. In other words, by moving the controller 7 away from themonitor 2, the player P can reduce, in size, the character image (imageIMs) displayed on the monitor 2. Moreover, similarly to the image IMb,by moving the controller 7 from side to side and up and down withrespect to the monitor 2 and/or twisting the controller 7 to the rightand left with respect to the monitor 2, the player P can move and/orrotate the image IMs displayed on the monitor 2. In FIG. 13, a positionof the image IMs, which is set in accordance with a position which theplayer P points with the controller 7, is shown as an image positionPims.

As is clear from a comparison of FIGS. 12 and 13, areas within which theimage IMb and image IMs can be respectively moved (hereinafter, theseareas are each referred to as a movable area) are set to have differentsizes from each other. Here, the movable areas of the image IMb andimage IMs are areas within which the image positions Pimb and Pims canbe set, respectively. To be specific, a relatively large movable area isset for the image IMb which is enlarged in display. In the example ofFIG. 12, a larger movable area than the display area of the monitor 2 isset. Also, a relatively small movable area is set for the image IMswhich is reduced in display. In the example of FIG. 13, a smallermovable area than the display area of the monitor 2 is set. By settingthe movable area, as described above, in accordance with an enlargementratio of the image IM, it is prevented that the size of the movable areabecomes insufficient when the image IMb has a relatively highenlargement ratio, and that the image IMs having a relatively smallenlargement ratio disappears from the display area when the image IMsmoves. As a result, the movement of the image IM can be properlycontrolled.

A moving speed of the image IM, which moves in response to the player Phaving moved the controller 7 from side to side and up and down withrespect to the monitor 2, is changed in accordance with theaforementioned distance D. To be more specific, in the state B, i.e., inthe case where the player operates the controller 7 near the monitor 2,the moving speed of the image IM is relatively fast when the positionpointed by the controller 7 is moved. On the other hand, in the state C,i.e., in the case where the player operates controller 7 at a remoteposition from the monitor 2, the moving speed of the image IM isrelatively slow when the position pointed by the controller 7 is moved.

Here, even if the player P moves the controller 7 from side to side at asame angle, a position pointed by the controller 7 varies in accordancewith a distance from the monitor 2 to the controller 7. For example, asshown in FIG. 14, when the player P is positioned near the monitor 2,the player P is required to move a controller 7N by an angle θN in orderto point both ends of the display screen with the controller 7N.Whereas, when the player P is positioned at a remote position from themonitor 2, the player P is required to move a controller 7F by an angleθF in order to point the both ends of the display screen with thecontroller 7F. When these two situations are compared, an operationangle θN is greater than an operation angle θF as shown in FIG. 14. Inother words, when the player P near the monitor 2 points the monitor 2with the controller 7, the player P is required to increase an angle bywhich to move the controller 7, as compared to a case where the player Poperates the controller 7 at a remote position from the monitor 2. Inorder to reduce such a difference in operability which is affected by achange in the distance D, a moving amount, of the image position Pimwhich is set based on a position pointed by the controller 7, is changedin accordance with the distance D. Note that, the image position Pim ofthe image IM, which is set based on the position pointed by thecontroller 7, is a placement position of the present invention.

Next, the game processing performed by the game system 1 will bedescribed in detail. First, main data used for the game processing willbe described with reference to FIG. 15. FIG. 15 shows an example of themain data to be stored in the external main memory 12 and/or internalmain memory 35 of the game apparatus body 5 (hereinafter, these two mainmemories are collectively referred to as a main memory).

As shown in FIG. 15, the main memory 33 stores operation information Da,operation status information Db, display information Dc and so on. Inaddition to data contained in the information shown in FIG. 15, the mainmemory 33 stores, as necessary, other data used for the game processing.

The operation information Da contains the series of pieces of operationinformation (key data, acceleration data and process result data) whichare transmitted as transmission data from the controller 7. Theoperation information Da is information to be updated to latestoperation information. The operation information Da contains firstcoordinate data Da1 and second coordinate data Da2 which correspond tothe positional information of the above-described process result data.For the image taken by the image pickup element 743, the firstcoordinate data Da1 indicates a position (position in the taken image)of one of the images of the two markers 8L and 8R, and the secondcoordinate data Da2 indicates a position (position in the taken image)of the other of the images of the two markers 8L and 8R. The positionsof the images of the markers in the taken image are specified, forexample, in a XY coordinate system on the taken image.

The operation information Da contains, in addition to the coordinatedata (the first coordinate data Da1 and second coordinate data Da2)which is exemplary process result data obtained from the taken image,key data Da3 and the like obtained from the operation sections 72. Notethat, the wireless controller module 19 of the game apparatus body 5receives the series of pieces of operation information transmitted fromthe controller 7 at predetermined time intervals, e.g., every 5 ms, andstores the operation information in a buffer (not shown) of the wirelesscontroller module 19. Thereafter, most recently stored operationinformation in the buffer is read every frame (e.g., every 1/60 sec)which is a game processing interval, and the operation information Da ofthe main memory is updated.

The operation status information Db contains information about anoperation status of the controller 7, which operation status isdetermined based on the taken image. The operation status information Dbcontains data which is obtained from, e.g., positions and directions ofthe target images (markers) contained in the taken image. To bespecific, the operation status information Db contains direction dataDb1, middle point data Db2, current distance data Db3, referencedistance data Db4, designated coordinate data Db5, designated directiondata Db6 and so on. The direction data Db1 indicates a direction from apoint indicated by the first coordinate data Da1 to a point indicated bythe second coordinate data Da2. It is assumed here that the directiondata Db1 is a vector whose originating point is the point indicated bythe first coordinate data Da1 and whose ending point is the pointindicated by the second coordinate data Da2. The middle point data Db2indicates coordinates of a middle point between the point indicated bythe first coordinate data Da1 and the point indicated by the secondcoordinate data Da2. When the images of the two markers (markers 8L and8R) are seen as a single target image, the middle point data Db2indicates a position of the single target image in the taken image. Thecurrent distance data Db3 indicates the current distance D between thecontroller 7 and the markers 8L and 8R and which is calculated based onthe first coordinate data Da1 and second coordinate data Da2. Thereference distance data Db4 indicates the distance D, which is adistance between the controller 7 and the markers 8L and 8R and which isobtained at a predetermined timing (e.g., a timing at which alater-described process starts), as the reference distance Dbs. Thedesignated coordinate data Db5 indicates a designated position in avirtual game space (Xt, Yt, Zt), which is designated by the controller7. The designated direction data Db6 indicates a designated direction inthe virtual game space, which is set based on an orientation of thecontroller 7 (e.g., angle of a twisting direction).

The display information Dc contains image scale value data Dc1, movingamount scale value data Dc2, movable area data Dc3, image position dataDc4, image angle data Dc5, image data DC6, and the like. The image scalevalue data Dc1 indicates scale values (Sx, Sy) of the image IM, whichare set based on the distance D and reference distance Dbs. The movingamount scale value data Dc2 indicates scale values (STx, STy) of amoving amount of the image IM, which are set based on the scale values(Sx, Sy). The movable area data Dc3 indicates an area within the virtualgame space, in which area an image position Pim is placeable. The imageposition data Dc4 indicates coordinates of the image position Pim (Xans,Yans, Zans) in the virtual game space. The image angle data Dc5indicates an image angle Aim which is used when the image IM is placedin the virtual game space. The image data Dc6 is image data fordisplaying the image IM, a background image and the like on a displaydevice (monitor 2).

Next, image processing performed by the game apparatus body 5 will bedescribed in detail with reference to FIGS. 16 to 19. FIG. 16 is aflowchart showing a sequence of the image processing performed by thegame apparatus body 5. FIG. 17 is a subroutine showing in detail adistance calculation process at steps 52 and 65 of FIG. 16. FIG. 18 is adiagram for describing an operation to calculate the distance D. FIG. 19is a conceptual diagram showing an image position and a movable area setin the virtual game space. Note that, flowcharts of FIGS. 16 and 17mainly illustrate, among a plurality of processes performed in the gameprocessing, a process for displaying the image IM on the monitor 2, anddetailed descriptions of the other processes which are not directlyrelevant to the present invention will be omitted. In FIGS. 16 and 17,each step performed by the CPU 10 is abbreviated as “S”.

When the power button 24 of the game apparatus body 5 is turned on, theCPU 10 of the game apparatus body 5 executes the boot program stored inthe ROM/RTC 13, thereby initializing each unit such as the main memory.Then, the game program stored in the optical disc 4 or other storagemedium is loaded to the main memory, whereby the game program becomesready to be executed by the CPU 10. The flowchart shown in FIG. 16 showsimage processing which is performed after this process is completed.

As shown in FIG. 16, the CPU 10 obtains the operation informationreceived from the controller 7 (step 50), and then proceeds to a nextstep. Next, the CPU 10 updates the operation information Da by using themost recently obtained operation information. The operation informationobtained at step 50 contains, in addition to the process result dataindicating the positions of the markers 8L and 8R in the taken image,key data indicating a manner in which an operation section 72 of thecontroller 7 has been operated. Here, the communication section 75transmits the operation information to the game apparatus body 5 atpredetermined time intervals (e.g., every 5 ms). The CPU 10 uses, ateach frame, the most recently transmitted operation information toupdate the first coordinate data Da1, second coordinate data Da2 and keydata Da3.

Next, the CPU 10 determines whether or not to start predetermined imageprocessing (step 51). For example, the CPU 10 determines to start theimage processing by referring to the key data Da3 and detecting that anoption to start the image processing has been selected or that a buttonto start the image processing has been pressed. In the case of startingthe image processing, the CPU 10 proceeds to a next step S52, whereas inthe case of not starting the image processing, the CPU 10 returns tostep 50, and reiterates the processes.

At step 52, the CPU 10 performs the distance calculation process, andthen proceeds to a next step. In the distance calculation process, thedistance D between the controller 7 and the markers 8L and 8R iscalculated based on the first coordinate data Da1 and second coordinatedata Da2, which have been transmitted from the controller 7 and storedin the main memory 33. Hereinafter, the distance calculation process atstep 52 will be described in detail with reference to FIGS. 17 and 18.

As shown in FIG. 17, the CPU 10 refers to the first coordinate data Da1and second coordinate data Da2 (step 71), and calculates a distance mi(step 72). As shown in FIG. 18, the distance mi is a distance betweentwo points in an image taken by the imaging information calculationsection 74 of the controller 7. These two points correspond to images ofthe markers 8L and 8R in the taken image, and data indicatingcoordinates of the two points are contained in the first coordinate dataDa1 and second coordinate data Da2. Accordingly, the CPU 10 cancalculate the distance mi by using the first coordinate data Da1 andsecond coordinate data Da2. To be specific, when the first coordinatedata Da1 are position coordinates (Lx, Ly) and the second coordinatedata Da2 are position coordinates (Rx, Ry), the distance mi iscalculated by the following equation.

mi=√{square root over ((Rx−Lx)²+(Ry−Ly)²)}{square root over((Rx−Lx)²+(Ry−Ly)²)}

Next, the CPU 10 calculates a width w (refer to FIG. 18) whichindicates, with respect to setting positions of the markers 8L and 8R, awidth for which the image pickup element 743 is able to take an image(step 73). The width w is obtained by the following equation.

w=wi×m/mi

Here, m represents a setting distance between the markers 8L and 8R(actual setting distance between the markers 8L and 8R; e.g., 20 cm),and is a fixed value. Also, wi represents a width wi of the image takenby the image pickup element 743, the width wi corresponding to the widthw. The width wi is also a fixed value. Since the setting distance m andwidth wi are fixed values, these values are prestored in storage meanswithin the game apparatus body 5. Note that, the player is allowed todiscretionarily determine the setting positions of the markers 8L and 8Rin accordance with the player's environment, thereby determining thesetting distance m. In such a case, if the player inputs, as the settingdistance m, a distance between the discretionarily determined settingpositions of the markers 8L and 8R, the above equation can be used inthe same manner as described above.

Next, the CPU 10 calculates the current distance D (refer to FIG. 18)between the image pickup element 743 (controller 7) and the markers 8Land 8R, by using the width w and a viewing angle θ of the image pickupelement 743, and updates the current distance data Db3 (step 74). Then,the distance calculation process in the subroutine ends. Here, thecurrent distance D is calculated by the following equation.

D=(w/2)/{tan(θ/2)}

Since the viewing angle θ is a fixed angle, the angle θ is prestored inthe storage means within the game apparatus body 5.

Return to FIG. 16. After performing the process for calculating thedistance D at step 52, the CPU 10 sets, as the reference distance Dbs,the distance D stored in the current distance data Db3, updates thereference distance data Db4 (step 53), and then proceeds to a next step.In other words, the reference distance Dbs is the distance from thecontroller 7 to the markers 8L and 8R at the start of image processing.

Next, the CPU 10 calculates designated coordinates and a designateddirection (step 54), and proceeds to a next step. Hereinafter, anexemplary manner of calculating the designated coordinates anddesignated direction will be described.

For example, the CPU 10 calculates a direction from the positioncoordinates (Lx, Ly) indicated by the first coordinate data Da1 to theposition coordinates (Rx, Ry) indicated by the second coordinate dataDa2, and a middle point between the position coordinates (Lx, Ly) andthe position coordinates (Rx, Ry). Then, the CPU 10 updates thedirection data Db1 and middle point data Db2. To be specific, the CPU 10refers to the first coordinate data Da1 and second coordinate data Da2,thereby calculating coordinates of the middle point, and then updatesthe middle point data Db2. Here, when the target images (markers 8L and8R) in the taken image are seen as a single image, the middle pointindicates a position of the single image. A change in the position ofthe image, which change occurs in response to the controller 7 havingchanged a position thereof with respect to the monitor 2, can becalculated based on a difference between the middle point and apredetermined reference position.

Described below is a positional relationship among the controller 7, thedisplay screen of the monitor 2, and the markers 8L and 8R. For example,described below is a case where the markers 8L and 8R are provided on atop surface of the monitor 2 (see FIG. 1), and the player points at thecenter of the display screen of the monitor 2 by using the controller 7whose top surface is facing in an upper direction (i.e., a case where animage of the center of the display screen is present at the center of animage taken by the imaging information calculation section 74). At thispoint, in the image taken by the imaging information calculation section74, a middle point between the target images (middle point between themarkers 8L and 8R) does not coincide with the center of the taken image.To be specific, the target images in the taken image are in upperpositions than the center of the taken image. Here, when the targetimages are in such positions in the taken image, the positions are setas reference positions, and the center of the display screen is assumedto be pointed by the controller 7. When the controller 7 moves, thepositions of the target images in the taken image move accordingly (amoving direction of the target images is opposite to that of thecontroller 7). For this reason, by performing processing for moving thepointed position on the display screen in accordance with a change inthe positions of the target images in the taken image, the position (xd,yd) pointed by the controller 7 can be calculated with respect to thedisplay screen. Here, the reference positions may be set such that theplayer points at a predetermined position on the display screenbeforehand, and the positions of the target images at the time may bestored as the reference positions associated with the predeterminedposition. Alternatively, if a positional relationship between thedisplay screen and the target images is fixed, the reference positionsmay be preset. The above position coordinates (xd, yd) with respect tothe display screen are calculated by linear conversion using a functionfor calculating, based on the middle point, coordinates with respect tothe display screen of the monitor 2. The function used herein is forconverting coordinate values of the middle point, which are calculatedfrom a particular taken image, to coordinate values (xd, yd)representing a position on the display screen, which position has beenpointed by the controller 7 when the taken image has been taken. Byusing this function, the pointed position (xd, yd) can be calculated,with respect to the display screen, from the middle point coordinates.Note that, in the case where the player points at the display screen ofthe monitor 2 by using the controller 7 whose top surface is facing inother direction than the upper direction (e.g., facing to the right),the middle point coordinates are corrected using the direction stored inthe direction data Db1, and then the pointed position (xd, yd) iscalculated as described above with respect to the display screen, byusing the corrected middle point coordinates.

Further, the calculated pointed position (xd, yd) with respect to thedisplay screen is converted by the CPU 10 to a corresponding position ina virtual game space, whereby designated coordinates (Xt, Yt, Zt) arecalculated, and the designated coordinates data Db 5 is updated. Forexample, when an XY coordinate system is set having the center of thedisplay screen as a coordinate origin as shown in FIG. 19, thedesignated coordinates (Xt, Yt, Zt) corresponding to the pointedposition (xd, yd) represent a position in the virtual game spacedisplayed on the display screen of the monitor 2 (e.g., a position whichis perspective-projected on a Z=Zt plane (moving plane) in the virtualgame space). To be specific, the CPU 10 calculates the designatedcoordinates (Xt, Yt, Zt) by using the equation below.

$\begin{pmatrix}{Xt} \\{Yt} \\{Zt}\end{pmatrix} = {\begin{pmatrix}\; \\{{PROJECTION}\mspace{14mu} {MATRIX}} \\\;\end{pmatrix}^{- 1} \times \begin{pmatrix}{xd} \\{yd} \\1\end{pmatrix}}$

Similarly, the CPU 10 converts the direction stored in the directiondata Db1 (vector data) to a corresponding designated direction in thevirtual game space, and updates the designated direction data Db6. Forexample, the CPU 10 converts the direction stored in the direction dataDb1 to a designated direction to be projected on the aforementionedmoving plane set in the virtual game space, and updates the designateddirection data Db6. This allows the player to intuitively determine anorientation of the image Im in accordance with an orientation of thecontroller 7.

Next, the CPU 10 sets scale values (Sx, Sy) of the image IM, and updatesthe image scale value data Dc1 (step 55), and proceeds to a next step.For example, the CPU 10 refers to the distance D contained in thecurrent distance data Db3 and the reference distance Dbs contained inthe reference distance data Db4, thereby calculating the scale values(Sx, Sy) based on the following equations.

Sx=2.0+(Dbs−D)/a1

Sy=Sx

Here, a1 is an invariable, e.g., a1=17.5. Note that, the scale values Sxand Sy which the CPU 10 takes are both in a range from 0.4 to 20.0. Tobe specific, when results of calculation by the above equations showthat the scale values Sx and Sy are both smaller than 0.4, the CPU 10sets both the scale values Sx and Sy to 0.4. Also, when results ofcalculation by the above equations show that the scale values Sx and Syare both greater than 20.0, the CPU 10 sets both the scale values Sx andSy to 20.0. As is clear from the above equations, the invariable a1 is avalue for adjusting sensitivity to change, in accordance with a changein the distance D, the scale values (Sx, Sy), and the value may be setto an arbitrary value in accordance with later-described sensitivity toenlarge/reduce the image IM.

Subsequently, the CPU 10 sets scale values (STx, STy) of a moving amountof the image IM, and updates the moving amount scale value data Dc2(step 56). Then, the processing proceeds to a next step. For example,the CPU 10 refers to the scale values (Sx, Sy) contained in the imagescale value data Dc1, and calculates the scale values (STx, STy) basedon the following equations.

STx=Sx/a2+a3

Sty=Sy/a4+a5

Here, a2 to a5 are invariables, e.g., a2=9.0, a3=1.0, a4=5.0 and a5=1.0.These invariables a2 to a5 are parameters used for naturally moving theimage IM on the display screen, and are set to arbitrary values inaccordance with, e.g., an aspect ratio of the monitor 2.

Then, the CPU 10 sets the movable area within the virtual game space,and updates the movable area data Dc3 (step 57). Then, the processingproceeds to a next step. For example, the CPU 10 refers to the scalevalues (STx, Sty) contained in the moving amount scale value data Dc2,and sets the movable area within which the image position Pim isplaceable. To be specific, the virtual game space has, as shown in FIG.19, the XY coordinate system set on the Z=Zt plane (moving plane), theXY coordinate system having a gazing point as a coordinate origin. Then,for a reference movable area, which is preset on the moving plane byhaving the aforementioned coordinate origin as a center thereof, the CPU10 multiplies the scale value STx by a length of the reference movablearea in an X-axis direction, and multiplies the scale value Sty by alength of the reference movable area in a Y-axis direction, therebycalculating a movable area which is a result of enlarging/reducing thereference movable area with respect to the above coordinate origin. Notethat, shown in the example of FIG. 19 is a movable area which is aresult of enlarging the reference movable area with respect to thecoordinate origin of the XY coordinate system.

Next, the CPU 10 sets the image position Pim, and updates the imageposition data Dc4 (step 58). Then, the processing proceeds to a nextstep. For example, the CPU 10 refers to the designated coordinates (Xt,Yt, Zt) contained in the designated coordinates data Db5 and the scalevalues (STx, Sty) contained in the moving amount scale value data Dc2,thereby calculating coordinates of the image position Pim (Xans, Yans,Zans) based on the following equations.

Xans=STx×Xt

Yans=STy×Yt

Zans=Zt

For example, the virtual game space has, as shown in FIG. 19, the XYcoordinate system set on the Z=Zt plane (moving plane), the XYcoordinate system having the gazing point as a coordinate origin. Then,the image position Pim is set on a position (Xans, Yans, Zans) on themoving plane, which position results from increasing/reducing, inaccordance with the scale values (STx, Sty), X-axis and Y-axiscoordinate values of the designated coordinates (Xt, Yt, Zt) on themoving plane, the designated coordinates corresponding to the position(xd, yd) pointed by the controller 7. In other words, the image positionPim is calculated based on the designated coordinates (Xt, Yt, Zt), andwhen the designated coordinates (Xt, Yt, Zt) move on the moving plane,the image position Pim moves on the moving plane by a moving amountresulting from increasing/reducing the moving distance of the designatedcoordinates in accordance with the scale values (STx, Sty).

Note that, in the case where the image position Pim, which is calculatedby the process at the above step 58, is set to be outside the movablearea set at the above step 57, the CPU 10 resets the image position Pimso as to be within the movable area, and then updates the image positiondata Dc4. As one example, when the image position Pim is set to beoutside the movable area, the CPU 10 resets the image position Pim so asto be at a boarder of the movable area, which is the nearest boarder tothe set image position Pim. As another example, when the image positionPim is set to be outside the movable area, the CPU 10 resets the imageposition Pim so as to be at an intersection point of a line, whichconnects the set image position Pim and the coordinate origin of the XYcoordinate system on the moving plane (i.e., the gazing point), and aboarder of the movable area. As a result of this process, the imageposition Pim is always placed within the movable area set at the abovestep 57.

Next, the CPU 10 sets the image angle Aim, and updates the image angledata Dc5 (step 59). Then, the processing proceeds to a next step. Forexample, based on the designated direction contained in the designateddirection data Db6, the CPU 10 calculates the image angle Aim whichindicates a display orientation of the image IM along the moving plane(e.g., upward orientation of the image IM).

Next, the CPU 10 enlarges/reduces the image IM in accordance with thescale values (Sx, Sy) (step 60), and proceeds to a next step. Forexample, the CPU 10 refers to image data contained in the image dataDc6, which image data indicates the image IM, and enlarges the image IMby an enlargement ratio Sx in a lateral direction and by an enlargementratio Sy in a longitudinal direction. Here, as is clear from the processat step 55, when the controller 7 is moved toward the markers 8L and 8Rso as to be at a shorter distance to the markers 8L and 8R than thereference distance Dbs, the scale values (Sx, Sy) are set to be greater,accordingly. Therefore, when the player moves the controller 7 towardthe monitor 2, the image IM is enlarged, whereas when the player movesthe controller 7 away from the monitor 2, the image IM is reduced.

Next, the CPU 10 places the image IM in the virtual game space inaccordance with the image position Pim and image angle Aim (step 61),and proceeds to a next step. To be specific, the CPU 10 places the imageIM, which has been enlarged/reduced at the above step 60, on the movingplane in accordance with the image position Pim and in an orientationcorresponding to the image angle Aim. For example, as shown in FIGS. 12and 13, the image IM is a human figure resembling the player, and theimage IM is placed on the moving plane such that the image position Pimis positioned at the center of the human figure (e.g., near the waist).Further, a placement angle of the image IM rotates around the imageposition Pim in accordance with the image angle Aim.

Next, the CPU 10 displays on the monitor 2 an image of the virtual gamespace seen from a predetermined viewpoint (see FIG. 19) (step 62). Then,the CPU 10 determines whether or not to terminate the image processingstarted at the above step 51 (step 63). When the CPU 10 continues theimage processing, the CPU 10 proceeds to a next step 64, whereas whenthe CPU 10 terminates the image processing, the CPU 10 ends theprocessing according to the flowchart.

At step 64, the CPU 10 obtains the operation information received fromthe controller 7. Then, the CPU 10 performs the distance calculationprocess (step 65), and returns to the above step 54 to reiterate theprocesses. Note that, since the operation performed at step 64 is thesame as that of the above-described step 50, and the operation performedat step 65 is the same as that of the above-described step 52, detaileddescriptions thereof will be omitted.

As described above, according to the game apparatus body 5 of thepresent embodiment, when the player moves the controller 7 toward themonitor 2, the displayed image IM is enlarged, and the area within whichthe image IM can be moved is also enlarged. On the other hand, when theplayer moves the controller 7 away from the monitor 2, the displayedimage IM is reduced, and the area within which the image IM can be movedis also reduced. Consequently, when the image IM has a high enlargementratio, the movable area is set to be large, whereby operabilitydeterioration, e.g., the movable area becoming insufficient in size forthe image having a high enlargement ratio, is prevented. Also, when theimage IM has a low enlargement ratio, i.e., when the displayed image IMis reduced, the movable area is reduced, whereby the image IM isprevented from disappearing from the display area.

Further, according to the game apparatus body 5 of the presentembodiment, the moving amount of the image position Pim is greater inthe case where the image position Pim is designated when the controller7 is positioned near the monitor 2 than in the case where the imageposition Pim is designated when the controller 7 is at a remote positionfrom the monitor 2. As a result, increase/decrease, caused by adifference in distance between the controller 7 and monitor 2, of themoving amount of the image position Pim in relation to an amount ofchange in the position pointed by the controller 7, is lessened. Inother words, for the player using the controller 7 to perform anoperation to move the pointed position, a significant change inoperability does not occur even if the distance between the controller 7and the monitor 2 changes. This allows the player to perform theoperation without feeling discomfort.

Still further, since the player can perform the imageenlargement/reduction process only by changing the distance between thecontroller 7 and monitor 2, the player is allowed to change theenlargement ratio of the displayed image by a simple operation. Also,the image IM displayed on the monitor 2 moves or rotates in response tothe controller 7 having been moved from side to side and up and downwith respect to the monitor 2 or having been twisted to the right andleft with respect to the monitor 2. Thus, the displayed image can bemoved/rotated by a simple operation.

The above description of the image processing gives an example in whichthe movable area is set in accordance with the enlargement ratio of theimage IM. However, the movable area may be set in accordance with, inaddition to the enlargement ratio, a shape of the image IM (e.g., aspectratio of the image) and placement angle of the image IM. For example,the aforementioned invariables a2 to a5 may be adjusted in accordancewith the aspect ratio of the image IM. This enables the aspect ratio ofthe movable area to be corrected in accordance with the aspect ratio ofthe image IM. Further, in addition to the aspect ratio of the image IM,the placement angle of the image IM may be taken into account to adjustthe invariables a2 to a5. For example, when a vertically long image IMis rotated by 90° and then placed, the image IM becomes a horizontallylong image. The shape of the image IM, which changes in accordance withthe placement angle, may be taken into account when changing theinvariables a2 to a5 in accordance with the aspect ratio and image angleAim of the image IM, whereby the aspect ratio of the movable area may becorrected.

Still further, in the above description of the image processing, theimage is small in size when the distance between the controller 7 andmonitor 2 is long, and large in size when the distance is short.However, this may be set in the opposite manner. In other words, thepresent invention is applicable even in the case where the image is setto be large when the distance between the controller 7 and monitor 2 islong, and set to be small when the distance is short. In such a case, asa specific example, the above step 55 may be performed based on thefollowing equation.

Sx=18.0+(D−Dbs)/a1

The present invention is not limited to controlling the enlargementratio of an image in accordance with the distance between the controller7 and monitor 2, but applicable to general image processing forenlarging, reducing and moving an image. For example, even if thepresent invention is applied to a process for placing an image atcoordinates inputted by a mouse and enlarging/reducing the image througha key operation or other operation, the effect of the present inventioncan be obtained by controlling, in accordance with the enlargement ratioof the image, a moving amount and moving range of the coordinates inrelation to a positional change of the mouse.

Still further, in the above description of the image processing, thescale values (Sx, Sy) of the image IM and the scale values (STx, STy) ofthe moving amount are set to be different from each other. This enablesseparate adjustment of the enlargement/reduction of the image and thecontrol of the moving amount, whereby the image IM is allowed to benaturally moved in accordance with the aspect ratio of the displayscreen while being enlarged/reduced longitudinally and laterally with asame enlargement ratio. However, when such effect is unnecessary, thescale values (STx, Sty) of the moving amount may be set to be the sameas the scale values (Sx, Sy) of the image IM.

Still further, the above description of the image processing gives anexample in which the image IM, which is placed in a three-dimensionalvirtual game space, is displayed. However, the image which is placed ina different type of game space may be displayed. It is understood thatthe present invention can be realized even for, e.g., image processingfor enlarging/reducing and moving/rotating a two-dimensional image IMplaced in a two-dimensional virtual game space.

Still further, it is described above that the processes for, e.g.,enlarging, reducing, moving and rotating the image in accordance withthe position and orientation of the controller 7 are performed at anytime from the start to the end of the image processing. However, theprocesses may be performed only when a predetermined operation isperformed on the controller 7. For example, only when the player ispressing a predetermined button of the controller 7 (e.g., operationbutton 72 i), the processes for, e.g., enlarging, reducing, moving androtating the image may be performed. In this case, the image processingmay be performed by setting the distance D, which is obtained when thepredetermined button starts to be pressed, as the reference distanceDbs.

In the above description, the distance D is calculated by analyzing theimage data of the image taken by the image pickup element 743. However,the above invention can be implemented as long as the distance D to apredetermined measuring target placed in a real space is measured in anymanner. For example, a supersonic sensor or a magnetic sensor may beused as means of calculating the distance D. The distance D may notnecessarily directly be calculated if there is any manner to obtain avalue related to a distance between the image pickup element 743 and animaging target(s), because operation inputs can be performed as long assuch a value is obtained. In such a case, data corresponding to adistance between the markers 8L and 8R in the taken image may beprepared in advance, and by using the data, the image processing may beperformed.

Further, in the above description, the image data of the image taken bythe image pickup element 743 is analyzed to obtain the positioncoordinates, center of gravity coordinates and the like of the infraredlights emitted from the markers 8L and 8R. The process result dataindicating such coordinates and the like is generated by the controller7, and transmitted to the game apparatus body 5. However, data, whichcan be obtained in the middle of a process performed by the controller 7for generating the process result data, may be transmitted from thecontroller 7 to the game apparatus body 5. For example, the image dataof the image taken by the image pickup element 743 may be transmittedfrom the controller 7 to the game apparatus body 5, and the processresult data may be obtained as a result of analyzing the image data atthe CPU 10. In this case, the image processing circuit 744 providedwithin the controller 7 is no longer necessary. Alternatively, data as aresult of partly analyzing the image data may be transmitted from thecontroller 7 to the game apparatus body 5. For example, data indicatinga brightness, position, square measure and the like obtained from partlyperforming the analysis of the image data may be transmitted from thecontroller 7 to the game apparatus body 5, and the rest of the analysismay be performed by the CPU 10 to obtain the process result data.

Still further, in the above description, the infrared lights from themarkers 8L and 8R are the imaging targets of the imaging informationcalculation section 74 of the controller 7. However, a different objectmay be used as an imaging target. For example, one or more than threemarkers may be placed in the vicinity of the monitor 2, and an infraredlight(s) emitted therefrom may be used as an imaging target(s) of theimaging information calculation section 74. For example, the presentinvention may be realized by placing near the monitor 2 a single markerhaving a predetermined length between both ends thereof, and using thepredetermined length as the setting distance m (see FIG. 18).Alternatively, the display screen of the monitor 2 or another illuminant(e.g., interior light) may be used as the imaging target of the imaginginformation calculation section 74. Various illuminants can be used asimaging targets of the imaging information calculation section 74, byusing a manner of calculating, based on a positional relationshipbetween an imaging target and the display screen of the monitor 2, aposition of the controller 7 in relation to the display screen of themonitor 2.

Still further, in the above description, the controller 7 and the gameapparatus body 5 are connected by radio communication. However, thecontroller 7 and game apparatus body 5 may be electrically connected bya cable. In such a case, the cable connected to the controller 7 isconnected to a connecting terminal of the game apparatus body 5.

It is understood that the shapes of the controller 7 and the operationsections 72 mounted thereon, the number of operation sections 72, thepositions in which the operation sections 72 are provided, the equationsand invariables used for the image processing, and the like in the abovedescription are merely examples. The present invention can be realizedeven if these shapes, numbers, positions, equations, invariables and thelike are different from the above description. Also, the position of theimaging information calculation section 74 (the position of the lightentrance of the imaging information calculation section 74) of thecontroller 7 is not necessarily on the front surface of the housing 71.The imaging information calculation section 74 may be provided on anyother surface of the housing 71 as long as the image calculation section74 is able to externally receive a light.

Although the game apparatus body 5 is operated by the controller 7 inthe present embodiment, the game apparatus body 5 may be a generalinformation processing apparatus such as a personal computer which isoperated by an input device having image pickup means. In such a case, aprogram to be executed by a computer of the information processingapparatus is not limited to a typical game program used for a game. Theprogram to be executed may be an all-purpose image processing programwhich is used for image processing by the information processingapparatus.

The image processing program of the present invention may be supplied tothe game apparatus body 5 not only via an external storage medium suchas the optical disc 4, but also via a wired or wireless communicationline. Further, the image processing program may be prestored in anon-volatile storage device provided within the game apparatus body 5.Note that, an information storage medium for storing the imageprocessing program may be a CD-ROM, DVD or other similar optical discstorage medium, or may be a non-volatile semiconductor memory.

The storage medium storing the image processing program and the imageprocessing apparatus according to the present invention are capable ofproperly controlling the movement of an image in accordance with theenlargement ratio of the image or the distance between the image-pickupdevice and the imaging targets, and are useful as a program and anapparatus for performing various image processing such as gameprocessing.

While the invention has been described in detail, the foregoingdescription is in all aspects illustrative and not restrictive. It isunderstood that numerous other modifications and variations can bedevised without departing from the scope of the invention.

1-9. (canceled)
 10. A storage medium storing an image processing programto be executed by a computer of an image processing apparatus forplacing a displayed image on a predetermined placement position in adisplay area in accordance with a designation input obtained from aninput device having an image-pickup configured to take an image of animaging target, the image processing program comprising instructionscausing the computer to: obtain the designation input from the inputdevice; calculate, based on a position of the imaging target in thetaken image, a distance between the imaging target and the image-pickup;based on the distance, change a rate of change of the placement positionin relation to a change of the position of the imaging target in thetaken image, thereby calculating the placement position; placing theimage on the placement position; and causing a display device to displaythe image at the image placement position, wherein the placementposition calculation instructions cause the computer to calculate theplacement position in such a manner as to increase the rate of change ofthe placement position in accordance with a decrease in the distancecalculated by the distance calculation.
 11. The storage medium storingthe image processing program according to claim 10, further causing thecomputer to function as an area setter for, in accordance with thedistance calculated by the distance calculation, changing, in relationto the display area, an area within which the placement position isallowed to be set, wherein the area setting sets the area so as toextend in accordance with a decrease in the distance calculated by thedistance calculation.
 12. The storage medium storing the imageprocessing program according to claim 10, wherein a plurality of imagingtargets are provided, and the distance calculation calculates a distancebetween the image-pickup and the imaging targets, based on a distancebetween the imaging targets in the taken image. 13-16. (canceled)
 17. Animage processing apparatus for placing an image on a predetermindedplacement position in relation to a display area of a display device inaccordance with a designation input obtained from an input device havingan image-pickup for taking an image of an imaging target, the imageprocessing apparatus comprising a processor configured to: obtain thedesignation, input from the input device; calculate a distance, based ona position of the imaging target in the taken image, between the imagingtarget and the image-pickup; based on the calculated distance, change arate of change of the placement position in relation to a change of theposition of the imaging target in the taken image, thereby calculatingthe placement position; place the image on the placement position; andcausing a display device to display the placed image, wherein theplacement position calculation calculates the placement position in sucha manner as to increase the rate of change of the placement position inaccordance with a decrease in the distance calculated by the distancecalculation.
 18. The image processing apparatus according to claim 17,wherein the processor is further configured to, in accordance with thedistance calculated by the distance calculation change, in relation tothe display area, an area within which the placement position is allowedto be set, and set the area so as to extend in accordance with adecrease in the distance calculated by the distance calculation.
 19. Inan interactive display system for use with a display device and ahandheld imaging device that images an imaging target, the displaysystem comprising a processor that receives information from thehandheld imaging device and is configured to enable a user to displayand move an object on the display device in accordance with changes inwhere the input device is pointing, a method comprising: (a) using theprocessor to automatically determine where to display the object basedon information obtained by the handheld imaging device; (b) using theprocessor to automatically determine, based on optical informationobtained by the handheld imaging device, a change in distance of thehandheld imaging device from the display device; (c) using the processorto calculate an object image movement scaling value specifying the ratethe placement position of a displayed object changes with change in theinput device pointing direction so as to increase the rate of change ofthe placement position in accordance with a decrease in the distance ofthe handheld imaging device from the display device; and (d) using theprocessor to automatically control how the displayed object moves on thedisplay device in response to movement of the input device based atleast in part on the calculated object image movement scaling value. 20.In an interactive display system for use with a display device and ahandheld imaging device that images an imaging target, the displaysystem comprising a processor that receives information from thehandheld imaging device and is configured to enable a user to displayand move an object on the display device in accordance with changes inwhere the input device is pointing, a method comprising: (a) using theprocessor to automatically determine where to display the object basedon information obtained from the handheld imaging device; (b) using theprocessor to automatically determine, based on optical information, achange in distance between the handheld imaging device and the displaydevice; (c) using the processor to calculate an object image movementscaling value specifying the rate the placement position of a displayedobject changes with change in the input device pointing direction so asto increase the rate of change of the placement position in accordancewith a decrease in the distance; and (d) using the processor toautomatically control how the displayed object responds to movement ofthe input device based at least in part on the calculated object imagemovement scaling value.
 21. A non-transitory computer readable storagemedium having stored thereon an image processing program to be executedby a computer which displays a predetermined object image at apredetermined placement position in a display area on a display devicein accordance with information obtained from an input device, the inputdevice having an image-pickup device/sensor for taking images of animaging target in a pointing direction of the input device, and formoving and displaying the predetermined object image at a determinedplacement position in the display area in accordance with changes in aposition of the image of the imaging target in the image-pick up devicecorresponding to a movement of a pointing direction of the input device,the image processing program causing the computer to perform operationscomprising: determining a designation input based on informationobtained from the input device and calculating the predeterminedplacement position based at least in part on the designation input;calculating a placement position in accordance with changes in aposition of the image of the imaging target in the image-pick up device,wherein the placement position is calculated using a user-settable imageobject scale value, using the user-settable image object scale value toaffects a rate of change of a placement position with respect to amovement in the position of the imaging target in the taken image in amanner that causes an increase in the rate of change of the placementposition in accordance with a decrease in the calculated distancebetween the imaging target and the image-pickup device; displaying theimage at the calculated placement position; and causing a display deviceto display the object image at the calculated placement position on thedisplay device.
 22. An interactive display system for use with a displaydevice and an associated imaging target, the display system comprising:a handheld input device having an image-pickup for capturing views ofthe imaging target in a pointing direction of the input device; agraphics processor configured to display an object at a placementposition on the display device; and a processor that receivesinformation from the input device image-pickup and is operativelycoupled to the graphics processor, the processor being configured toenable a user to perform object placement setting for displaying andmoving the object on the display device in accordance with changes inthe pointing direction of the input device, the processor being used to:(a) automatically determine an object placement position based oninformation obtained from the input device image-pick up; (b) based oninformation obtained from the input device image pickup, automaticallydetermine a change in at least one aspect of distance of the inputdevice from the imaging target as the user moves the input device towardor away from the display device; (c) automatically controlling thegraphics processor to generate a movable object to be displayed on thedisplay device by using an object image size scale value based at leastin part on the change in distance of the input device from the imagingtarget; (d) using the object image size scale value to calculate anobject image movement scaling value specifying the rate the placementposition of the displayed movable object changes with change in theinput device pointing direction; and (e) using the object image sizescale value and the object image movement scaling value to automaticallycontrol how the displayed object responds to movement of the inputdevice.
 23. The display system of claim 22 wherein the processor setsthe object image size scale value based on user movement of theimage-pickup device toward or away from the display device during animage object position setting process.
 24. The system of claim 22wherein the input device includes at least one user-depressible control,and the processor is connected to operatively receive an indication whenthe user depresses the user-depressible control and sets the objectimage size scale value scale value in response to said user depressionof the user-depressible control.
 25. The system of claim 22 wherein theprocessor uses the object image size scale value to calculate an objectimage movement scaling value thereby increasing the rate of change ofthe placement position in accordance with a decrease in distance betweenthe image pickup and the imaging target.
 26. A non-transitory computerreadable storage medium having stored thereon an image processingprogram for performing an object image setting process to be executed bya computer operatively connected to a display device, the object imagesetting process enabling user control of movement, placement and displayof a predetermined two-dimensional object image at a user designatedplacement position within a set display area in accordance with positiondesignation information obtained from an input device, the input devicehaving an image-pickup device for taking an image of an imaging target,computing an updated placement position in accordance with movements ofan image of the imaging target acquired by the image-pickup device, anddisplaying the predetermined image at the updated placement position,the image processing program causing the computer to perform operationscomprising: initiating an object image setting process in response to auser input; determining a designation input based on informationobtained from the input device and determining a placement position ofthe object image in the display area based at least in part on thedesignation input, wherein the placement position is changed at apredetermined rate in relation to a movement/change in a position of theimaging target in the taken image; determining a change in at least oneaspect of a distance between the display device and the image-pickupdevice, the change based upon a user movement of the input device towardor away from the display device during the initiated object imagesetting process; calculating an object image size scale value based atleast in part on the determined change in the aspect of the distance;using the object image size scale value to calculate an object imagemovement scaling value specifying a rate of change of a placementposition with respect to a change of a position of an image of theimaging target acquired by the image-pickup device; using the objectimage movement scaling value for determining an updated placementposition of the object image in the display area in accordance withmovements of an image of the imaging target acquired by the image-pickupdevice; placing the object image at the updated placement positionwithin the display area; and causing the display device to display theobject image at the updated placement position.
 27. An interactivedisplay system for use with a display device and a handheld imagingdevice that images an imaging target, the display system comprising aprocessor that receives information from the handheld imaging device andis configured to enable a user to display and move an object on thedisplay device in accordance with changes in where the input device ispointing, the processor being used to: (a) automatically determine whereto display the object based on information obtained from the handheldimaging device; (b) automatically determine a change in distance betweenthe handheld imaging device and the display device; (c) calculating anobject image movement scaling value specifying the rate the placementposition of a displayed object changes with change in the input devicepointing direction so as to increase the rate of change of the placementposition in accordance with a decrease in the distance; and (d)automatically controlling how the displayed object responds to movementof the input device based at least in part on the calculated objectimage movement scaling value.